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I  The  Challenge 


How  do  you  replace 
the  hull  and  streamline 
the  structure 
while  the  ship  is 
steaming  at  full  speed 
in  rough  waters? 
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Value,  Architecture,  and  Performance 


Value  engineering  involves  understand  ng 
business  value  and  making  informed  IT 
decisions  that  maximize  value. 

♦  Enterprise  arch  tecture  puts 
adaptive  frameworks  n  place 
that  enable  rapid  response  at 
the  speed  of  today’s  bus  ness. 


Enterprise 
Architecture 


Performance 

Engineering 


Performance  engineering  means  measuring 
position,  speed,  and  rate  of  change  and 
be  ng  ready  and  able  to  act  on  those 

measures.  BtEM&l 
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I  What  Should  We  Measure? 


“Would  you  tell  me,  please,  which  way  I  ought  to  go 
from  here?” 


“That  depends  a  good  deal  on  where  you  want  to 
get  to,”  said  the  Cat. 


“I  don’t  much  care  where--”  said  Alice. 


“Then  it  doesn’t  matter  which 
way  you  go,”  said  the  Cat. 

“--so  long  as  I  get  somewhere,” 
Alice  added  as  an  explanation. 

“Oh,  you’re  sure  to  do  that,” 
said  the  Cat,  “if  you  only  walk 
long  enough.” 
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Metrics  Purpose 


Provide  a  “moving  snapshot”  of  key  performance 
indicators 

•  Current  status 

•  Position  relative  to  targets 

•  Movement  toward  targets 

Provide  “alerts”  to  identify  critical  action  areas 

•  Focus  on  leading,  not  lagging  indicators 

•  Identification  of  expected  bands  of  acceptable 
performance 

Communicate  quantitative  aspects  of  organizational 
goals  in  terms  of  targets  and  performance 
improvement 
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Why  Measure? 


COMPARISON  LOCATION  DIRECTION 
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IT  /  Business  Alignment 


The  product  of  IT  is 
NOT  source  code.  It  is 
improved  business  performance. 

Understanding  and  coordinating 
IT  goals  and  measures  with 
Business  goals  and  measures 
is  essential. 


•  IT  processes  are  being  managed 
as  business  processes  that 
require  cost  /  benefit  analysis. 
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Measures  Must  Be  Clearly  Mapped 


Structural  Complexity 
Training  Profiles 
Staffing  Profiles 
Technology  Acceptance 


Organization,  Culture, 
Skills 


Architectural  Characteristics 
Level  of  Integration 
Level  of  Standardization 
Degree  of  Flexibility 
Quality  Support 
Reliability 

Architecture  & 
Support 


Team  Profiles 

Hierarchy  Measures 
Stability 

Experience  and  Training 
Productivity  vs.  Quality 
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Investment 


IT  Investment 
ROI 

Budget  Control 
Risk 

Performance  Profiles 
Portfolio  Investment 


Technology 


Change  Profile 
Technical  Condition 
Complexity 
Performance 
Reusability 

Size 

Quality 


Motivation 


Change  (Modification) 
Average  Time  to  Implement 
Frequency  of  Releases 
Cost  per  Modification 
Rework  Profiles 


IT  Processes  and  Projects 


Cost 

IT  Resources 
Staff/Project  Schedule 
Process  Quality 
Reliability  —  MTTF 
Customer  Responsiveness 
Time  to  Market 
Review  Profiles 
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Meaningful  Levels  of  Measurement 


Executive 


Decision  View 
-ROI 

-Business  Impact 

-  Price-performance 

-  Risk/Opportunity... 


Manager 


Management  View 

-  Costs/Budget 

-  Schedule/Effort/Delay 

-  Standards 

-  Resource  Availability... 


Operational  View 

-  Process/Activities 

-  Products/Specs 

-  Policy/Procedures 
Operations  -Constraints/Guides... 


1st  Qtr  2nd  Qtr  3rd  Qtr  4th  Qtr 


PARTNERSHIP  INNOVATION  RESULTS 


©2010EM&I 


Measurement  for  Management  Levels 


j 


Project 

Management 

Process 

Management 

Product 

Management 


y  yi 


Business 

Strategy 
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Metrics  Warehouse:  From  Measures  to  Strategy 


Effort 

Schedule 

Quality 

Level  1 

Quality 

Level  2 


People 

Project 

Process 

Product 

Team  size, 

#  application  users 

Total  hrs  budgeted, 
Total  $  budgeted 

#  steps 

KSLOC,  (FP), 

Pages  document 

Labor  rate 

%  effort  completed, 
Cost  variance 
(planned  vs.  actual) 

cost  by  CAPP, 
effort  by  CAPP, 

Defect  cost 

$/KSLOC,  $/page, 
$/application  j 

Actual  hours 
Available  hours 

%  Time  completed, 
Schedule  variance 
(planned  vs.  actual) 

steps  completed, 
Reviews  completed/ 
Total  #  reviews, 

Tests  completed/ 
Total  #  tests,  etc. 

products  completed, 
Programs  coded/ 

Total  #  programs, 
Reqmts  traced/ 

Total  #  reqmts,  etc. 

Defect  correction 
rate,  MTTR 

#  of  requirement 
changes 

#  defects  per  process 
downtime,  MTTF 

#  defects  per  product, 
complexity 

Closed  requests/ 
per  report  period 
(productivity) 

Defects  delivered 
(effectiveness) 

Defects  discovered 
by  phase, 

Age  of  open  defects 

Customer  satisfaction 
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Measures  Change 


C 


j 


OPTIMIZED 


Process  Change  Management 
Tech.  Change  Management 


MANAGED 

Software  Quality  Management 
Quant.  Process  Management 


PROCESS 

LEVEL 


Defect  Causal  Analysis 
Technology  Insertion  Analysis 
Productivity 


DEFINED 


Peer  Reviews 

Software  Product  Engineering 
Integrated  Software  Management 
Training  Programs 
Organization  Process  Definition 
Organization  Process  Focus 


REPEATABLE 

Software  Configuration  Management 
Software  Quality  Assurance 
SW  Project  Tracking  and  Oversight 
Software  Project  Planning 
Requirements  Management 


PRODUCT 

LEVEL 


Quality  Metrics 
Reliability 
Design 
Volatility 


f  APPLICATION 
LEVEL 


Complexity 

Performance 

Cost 


QUALITATIVE 

LEVEL 


INITIAL 


Size 

Effort 

Defects  (categorization) 
Cycle  Time  (schedule) 
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Comparative  Analysis 


Current 

Baseline 


Benchmarks 
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Project  Histories, 
Other  Baselines 
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Choosing  the  Right  Measures 


Measures  must  be  ACTIONABLE. 
Identifiable  contribution  to 

•  Business  decisions 

•  IT  decisions 

Measures  must  be  practical  to  collect. 

Keep  the  number  of  metrics  SMALL. 

Understand  the  roll-up  of  measures  into 
measures  at  other  management  levels. 
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Indicators,  not  absolutes 
Only  as  good  as  the  supporting  data 
Must  be  understood  to  be  of  value 

•  Business  understanding 

•  IT  understanding 

Not  for  judging  individual  performance 

Cannot  identify,  explain,  or  predict 
everything 

Need  iterative  analysis  from  multiple 
viewpoints 

Avoid  direct  comparison  of  projects 
No  single  metric 
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•  CHARACTERISTICS 

UNIT  OF  MEASURE 

•  Size  Progress 

Reuse  Rework 

Counts  of  physical  code 

•  Effort  Cost 

Rework 

Resource  allocations 

Counts  of  staff  hours  expended 

•  Schedule 

Calendar  dates  tied  to 
milestones,  reviews  and  audits, 
deliverable  products 

•  Quality  Rework 

Readiness  for  delivery 
Improvement  trends 

2010  EM&I 

Counts  of  software  problems 
and  defects 

SsEM&l 
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Getting  Started  /  Making  Progress 


Priorities  within  a  project: 

•  Understand  the  data  you  are  getting  now 

•  Standardize  the  content  of  future 
measurement  reports 

•  Define  and  collect  the  additional  information 
you  need  for  project  planning  and  tracking 


Priorities  within  an  organization: 

•  Understand  historical  data  you  already  have 

•  Get  consistent  data  from  project  to  project 


Get  consistent  data  over  time 
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I  What  should  we  measure,  and  why? 


•  Secure  executive  sponsorship 

•  Identify  critical  stakeholders 

•  Conduct  facilitated  workshops: 

•  Business  area  leaders 

•  IT  leaders 

•  Evaluate  proposed  measures 

•  Develop  balanced  scorecards 

•  Focus  on  the  alignment  and 
translation  of  IT  and  Business 
objectives 

•  Make  it  visible 

•  Own  the  responsibility 

•  Tailor  the  reward  system 
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Lessons  Learned 
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Communication 
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Balance 
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For  More  Information 


Engineering  Management  &  Integration  Inc. 
455  Spring  Park  Place,  Suite  350 
Herndon,  VA  20170 
USA 

+1  (703)  742-0585 
www.em-i.com 
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Elliot  Chikofsky  +1  (781 )  272-0049 

EM&I 

75  Lexington  St  Elliot.Chikofsky@em-i.com 

Burlington,  MA  01803  Elliot.Chikofsky@dfas.mil 

USA 
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